Repository navigation
Resend once in full when a worker answer names an unknown capabilities digest - #358
Merged
Merged
Conversation
Same issue as durable-workflow/sdk-php#448. Worker.run() and run_until() raised once the client's three transport attempts failed on cluster discovery or registration, so a native Server answering 503 {"reason":"storage_unavailable"} during warm-up (or a refused connection, or an upstream 502/503/504) ended the worker. Discovery and registration are now repeated whole through transient failures with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s: refused/reset/timed-out connections, upstream 502/503/504 without a protocol body, 503 storage_unavailable/backend_unavailable, and 5xx/429 with retryable: true. retryable: false, 401/403, other 4xx and an incompatible Server stay fatal and are logged at error level before run() raises. Pollers, completions and worker heartbeats already rode out these failures and are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Coordinator decisions for #355: - An activity completion or failure report that hits a transient is resent with the shared backoff uniform(0, min(4, 0.25 * 2**(n-1))) s while the attempt's lease (renewed lease, else the claimed lease_expires_at) covers the next wait. A transient never becomes a failure report. When the lease runs out the resend stops with an error log and the Server settles the attempt at lease expiry. - A resend answered 409 stale_attempt, recorded false, same task/attempt /owner, activity/attempt/task status all completed is this worker's committed completion (observed on durableworkflow/server:3.17.0; same rule as PHP's isCommittedActivityCompletion). Any other stale_attempt is a closed attempt, never success. - Poll-time 401/403 keep retrying so credentials can rotate, but every refusal is now logged at ERROR. The cross-SDK difference (PHP and Rust stop) is documented. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…RROR Review follow-ups for #355: - Poll retry waits are uniform(0, min(10, 0.25 * 2**(n-1))) s (the shared full jitter, capped at 10 s instead of growing to 30 s), still honouring a bounded Server retry delay. - A poller failing continuously for 60 s logs the outage at ERROR, then every 5 minutes while it lasts; a successful poll resets it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Digest re-review (MEDIUM): with garbled digests 0 of 10 workflows finished in 150 s, because an unknown digest raised ServerCapabilitiesDigestError and the leased task waited for its lease. As in PHP and Rust: on an unknown digest in a success answer, forget the held pair and resend that request once with "none" to get the full capabilities. Worker requests are fenced, so a completion the Server had already committed is answered as a duplicate (409 stale_attempt). An unknown digest on the resend (or after sending "none") is a protocol error. In an error body the field is dropped and the real HTTP error surfaces unchanged, without a resend. Co-Authored-By: Claude Opus 5.5 <[email protected]>
The duplicate-completion check runs when the committed-completion rule (#355) is present; it logs at INFO, so the test captures that level. Verified against this branch merged with #355. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Keeps both: this branch's transient retries, committed-completion rule and outage logs, and main's decoded response cap and continue-as-new result. Only CHANGELOG.md conflicted; both entries are kept. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Brings in the committed-completion rule, so the end-to-end test's garbled completion resend (409 stale_attempt) is accepted as success. Only CHANGELOG.md conflicted; both entries are kept. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Digest re-review principle for all SDKs: a resent completion can be answered as a duplicate without what the first answer carried (PHP lost leased activities this way). On an unknown digest in a success answer: - a non-poll request (completion, failure, heartbeat) is not resent: its answer is kept and the unknown digest is replaced by worker_protocol.server_capabilities from a read-only GET /api/cluster/info (dropped if discovery is unavailable), and the held pair is forgotten so later requests send "none"; - only task polls, safe to repeat, are resent once with "none". Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
rmcdaniel
added a commit
that referenced
this pull request
Nov 20, 2026
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Follow-up from the #354 digest re-review (MEDIUM), matching the PHP and Rust SDKs.
Problem
With garbled digests, 0 of 10 workflows finished within 150 s. An answer that named a digest the worker did not hold raised
ServerCapabilitiesDigestError, and the leased task then sat until its lease expired.Change (
Client._request,ServerCapabilitiesDigest)Principle: a non-idempotent request is never resent to recover capabilities. A resend can be answered as a duplicate that lacks what the first answer carried. PHP lost leased activities this way.
On an unknown digest in a success answer, the held pair is forgotten, so later requests send
noneand learn a fresh copy. Then:worker_protocol.server_capabilitiesfrom a read-onlyGET /api/cluster/info, which is the same object. If discovery is unavailable, the field is dropped; no caller reads capabilities from a non-poll answer.POST /worker/*-tasks/poll) are safe to repeat, so they are resent once withnone. An unknown digest on that resend, or in an answer to a request that already sentnone, raisesServerCapabilitiesDigestError.On an unknown digest in an error body, the field is dropped and the held pair forgotten. The real HTTP error surfaces unchanged and is not resent.
Python has no carried polls today. Even so, completions and heartbeats are never resent, so a future carried
activity_poll.answercannot be lost.Tests (
tests/test_capabilities_digest_unknown.py)drop_unknownremoves the field;forget()resets the header tonone.GET /cluster/info, sent without the digest header).none.none, and the next request sends the newly learned digestnoneWorker.run()):none, and the worker keeps pollingLocal results (Python 3.14):
pytest tests/ -m "not integration": 7891 passed, 3 skipped. This branch includes Keep the worker alive through transient runtime failures while it registers (sdk-php#448 parity) #355's branch (merged in) until Keep the worker alive through transient runtime failures while it registers (sdk-php#448 parity) #355 lands.ruff: clean.mypy: clean.🤖 Generated with Claude Code